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Abstract 

This research aims to produce mathematics problems based on PISA level with valid and 
practical content of change and relationships and has potential effect for Junior High School 
students. A development research method developed by Akker, Gravemeijer, McKenney and 
Nieveen is used this research. This development research consists of three stages; analysis, 
design, and evaluation. In the first stage, the researcher analyzed students, algebra material in 
school-based curricula (KTSP) and mathematics problems of PISA 2003 of change and 
relationships content. The second stage, the researcher designed 13 problems with content of 
change and relationships. The last, the researcher used formative evaluation design developed 
by Tessmer which includes self evaluation, one-to-one, expert review, small group, and field 
test. The data collect by walk through (step of expert review) to get valid problems in aspect of 
content, construct, and language; interview (steps of one-to-one, small group, and field test) to 
know problems practicality; questionnaire (step of field test) to know problems’ potential 
effects. The result of this research indicated that 12 mathematical problems based on PISA 
level of change and relationships content that developed have validity, practically, and 
potential effects for Junior High School students. 

Keywords: Development Research, Mathematical Problems Based On PISA Level, Change 
and Relationships. 


Abstrak 

Penelitian ini bertujuan untuk menghasilkan soal matematika model PISA konten change and 
relationships yang valid, praktis, dan memiliki efek potensial bagi siswa Sekolah Menengah 
Pertama (SMP). Adapun metode penelitian yang digunakan dalam penelitian ini adalah metode 
penelitian pengembangan yang dikembangkan oleh Akker, Gravemeijer, McKenney dan 
Nieveen. Penelitian pengembangan ini terdiri dari tiga tahapan yaitu analisis, desain, dan 
evaluasi. Pada tahap analisis, peneliti menganalisis siswa, menganalisis Kurikulum Tingkat 
Satuan Pendidikan (KTSP) materi aljabar, dan menganalisis soal matematika PISA 2003. Pada 
tahap desain, peneliti mendesain 13 butir soal matematika model PISA konten change and 
relationships. Selanjutnya, pada tahap evaluasi, peneliti menggunakan desain formative 
evaluation yang dikembangkan oleh Tessmer yang meliputi: self evaluation, one-to-one, expert 
review, small group, dan field test. Teknik pengumpulan data yang digunakan dalam penelitian 
ini adalah walk through (tahap expert review) untuk mengetahui validitas soal baik itu dari segi 
konten, konstruk, maupun bahasa; wawancara (tahap one-to-one, small group, dan field test) 
untuk mengetahui kepraktisan soal; dan angket (tahap field test) untuk mengetahui efek 
potensial soal. Adapun hasil penelitian ini menunjukkan bahwa peneliti menghasilkan 12 soal 
matematika model PISA konten change and relationships yang valid, praktis, dan memiliki 
efek potensial bagi siswa SMP. 

Kata kunci: Penelitian Pengembangan, Soal Matematika Model PISA, Change and 
Relationships. 


In PISA mathematics problems, there three prominent components; content, contexts, and 
competence, included in those problems. Content is a component in which materials are taught at 
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school. Context is a situational problem in daily life. A competence represents students’ ability to 
formulate, employ, and interpret mathematics in solving problems. 

Those three components could not be found in such integrative way in problems, for instance on 
Buku Sekolah Elektronik (BSE). On BSE, problems used still question abstract problems, formulas, 
and models which are far away from problems in PISA model. Problems given to students are 
expressed in mathematics language and symbol organized in contexts that are not close to daily 
problems. Furthermore, it can be said that recent problems used are not able to overlap students’ 
ability in solving problems in PISA. It results in students’ ability in solving problems of mathematics 
models in PISA that is said to be low due to students are not accustomed to solve those problems in 
their learning activity. 

In PISA 2003, algebra and measurement were significantly questions more difficult for 
Indonesia students than number, geometry, and data (Stacey, 2011: 118). It can be seen in the figure 
below, Indonesian score got lowest score in change and relationships conten than the other contents. 



Figure 1. Indonesian score in each content in PISA 2003 (Stacey, 2011: 217) 

Based on the figure above shows that, in PISA 2003, change and relationships content was 
getting the lowest score, 334. The other contents such as space and shape got 361, quantity 357, and 
uncertainty 385. Therefore, based on that results, the Indonesian student must be taught to do 
mathematical problems based on PISA level in learning, especially in the change and relationships 
content. 

PISA is study of literacy aimed to examine periodically the ability of 15-year-old students in 
reading literacy, mathematical literacy, and scientific literacy. PISA assesses the extent to which 15- 
year-old students can handle mathematics adeptly when confronted with situation and problems - the 
majority of which are presented in real-world contexts (OECD, 2000). The study of PISA was first 
conducted in 2000 and held every three years by Organisation for Economic Co-operation and 
Development (OECD). 

Students have mathematical literacy if students are able to formulate, implement, and interpret 
mathematics in avariety of contexts that include mathematical reasoning and using mathematical 
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concepts, procedures, facts and tools to describe, explain, and predict the phenomena in everyday life. 
This is explained in the framework of PISA 2012 as follows: 

Mathematical literacy is an individual’s capacity to formulate, employ, and interpret 
mathematics in a variety of contexts. It includes reasoning mathematically and using 
mathematical concepts, procedures, facts, and tools to describe, explain, and predict 
phenomena. It assissts individuals to recognise the role that mathematics plays in the world 
and make the well-founded judgments and decisions needed by constructive, engaged and 
reflective citizen (OECD, 2010). 

All of the competencies of students referred mathematization. According to De Lange (1987), 
mathematization is defined as the organization of activities in finding regularity, relations, and 
structures using the knowledge and skills early. The problems in PISA are based on the real world that 
students need to complete the process of mathematization to solve the real world problems 
mathematically. Mathematising can be characterised as having five aspects: (1) starting with a 
problem situated in reality; (2) organising it according to mathematical concepts; (3) gradually 
trimming away the reality through processes such as making assumptions about which features of the 
problem are important, generalising and formalising (which promote the mathematical features of the 
situation and transform the real problem into a mathematical problem that faithfully represents the 
situation); (4) solving the mathematical problem; and (5) making sense of the mathematical solution in 
terms of real situation. 

The characteristics of the mathematical PISA question consists of three components: 

1. Mathematical content that is targeted for use in the assessment items, that is quantity; change and 
relationships; space and shape; uncertainty and data. 

2. The context in which the assessment items are located, that is personal, occupational, societal, 
scientific. 

3. The mathematical process that describe what individuals do to connect for the problem with the 
mathematics and thus solve the problem, and the capabilities that underlie those processes, that is 
reproduction, connection, reflection. 

Change and relationships related to algebra in Kurikulum Tingkat Satuan Pendidikan (KTSP).. 
Kaput (van de Walle, 2007) states that there are five forms of algebraic form: (1) generalization of 
arimmetic and mathematical patterns; (2) the use of symbols that are quite useful; (3) learning about 
the structure of the number system; (4) learning the pattern and functions; and (5) mathematical 
modeling process, which brings together four ideas above. According to OECD (2009), the skills 
needed to study the change and relationships is the ability to: (1) Presenting the change in the form of 
an easily understood; (2) understand the types of fundamental changes; (3) recognizing the type of 
change of an event; and (4) applying techniques of change in the real world. 



50 IndoMS-JME, Volume 5, No. 1, January 2014, pp. 47-56 


METHOD 

A development research method developed by Akker, Gravemeijer, McKenney and Nieveen is 
used this research. This development research consists of three stages; analysis, design, and evaluation 
(Akker, et al., 2006). Analysis stage conducted an analysis of students, curriculum, and change and 
relationships content. The second stage, design the product (mathematics problems based on PISA 
level on change and relationships conten). The last stage, the researchers used formative evaluation 
design (figure 2) which includes self evaluation, one-to-one, expert review, small group, and field test 
(Tessmer, 1993; Zulkardi, 2006). 


Low resistance to revision -► High resistance to re\hsion 



Figure 2. Formative evaluation design (Tessmer, 1993; Zulkardi, 2006) 


The successful criteria of this research were obtained in the form of mathematics problems 
based on PISA level on change and relationships content in valid, practice, and have potential effect 
for the students. The validity can be determined by looking at the results of the expert validation 
(expert review stage). The practicality of the problems can be known from the opinion of students and 
observations in one-to-one and small group stage. Practicality means easy to use, can be administered 
and interpreted properly, and also does not have a variery of meanings. The potential effects can be 
determined in field tests based on the student answer, questionnaire and interview results. The results 
of questionnaire and interview were analyzed whether students think about the effect it has potential to 
explore students' mathematics skills. The results of the students' answers at the field test were also 
used as material analysis to determine the potential effect. 

Research subjects were junior high school students grade IX of Madrasah Tsanawiyah (MTs.) 
Negeri 1 Palembang , SMP Negeri 17 Palembang , SMP Negeri 14 Palembang , and SMP Negeri 1 
Palembang. The research was conducted on October 14, 2012 to December 5, 2012. However, the 
data collection techniques in this research were: (1) walk through, it performed on expert review stage 
to get valid in aspect of content, construct, and language of the mathematics problems from the expert 
by e-tools (email, phone, and POS Indonesia ); (2) interview, it performed on one-to-one, small group, 
and field test stages to know mathematics problems practicality; and (3) questionnaire, it performed on 
field test stage to know the potential effects of the mathematics problems. 
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The data collected was analyzed using descriptive analysis method: (1) analyzing the walk 
through sheet based on the comments of experts in the expert review stage for getting the validity; (2) 
analyzing interview results in the one-to-one, small group, and field test stage for getting the 
practically and for knowing the potential effect; (3) analyzing the questionnaire sheet in the field test 
stage for knowing the potential effect; and (4) analyzing the student answers for knowing the potential 
effect. 

RESULT AND ANALYSIS 
Analysis Stage 

We conducted an analysis of students, curriculum and content of PISA about change and 
relationships. Analysis of students aims to know the students who have the high, medium, and low 
ability in mathematics. We analyzed 39 students: four students in one-to-one, six students in small 
group, and 29 students in field test stage. 

Then, the analysis of the curriculum aims to know what Standar Komepetensi ( SK) and 
Kompetensi Dasar ( KD ) of algebra in KTSP related to change and relationships in PISA. Based on the 
analyzed, we got 3 SK (2, 3, and 4) and KD (2.3; 3.4; and 4.5) in VII grade, whereas in VIII grade, we 
got 2 SK (1 and 2) and KD (1.1 and 2.3). 

The last of the analysis stage was analysis of change and relationships content of PISA. In the 
analysis, we got four characteristics of change and relationships in PISA: (1) Generalization of 
numerical and geometrical patterns and of the laws governing numerical relationships; (2) problem 
solving; (3) functional situations; and (4) modelling of physical and mathematical phenomena. 

Design stage 

We designed 13 mathematics problems based on PISA level on change and relationships. All of 
the problems designed based on the content (according to the characteristics of PISA, in accordance 
with the change and relationships, and in accordance with the level of ability of 15-year-old students); 
construct (rich with mathematical concepts, inviting further concept development, using the command 
prompt, or demanding answers unraveled, and there are clear instructions on how to do the problems); 
and language (according to EYD , the matter does not contain a double interpretation, limits of 
questions and answers clearly, and tables, pictures, and graphics on the matter functions and presented 
in a clear and legible). According to us, all of the problems were change and relationships content. 
Evaluation stage 

a. Self Evaluation 

We evaluated the questions that designed by ourselves. The purpose of this evaluation is to 
know what the questions have three characteristics of PISA (content, context, and language). There is 
no change on the problems that designed. The result of this stage namely prototype I. 

b. Expert Review 
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Expert review serves to determine the validity of the problem based on the content, construct, 
and language of prototype I. There are three expert who validating the prototype I: (1) Prof. Kaye 
Stacey (The University of Melbourne, Australia); (2) Prof. Dr. Sutarto Hadi, M.Si., M.Sc. (Lambung 
Mangkurat University, Banjarmasin); and (3) Prof. Dr. M. Salman A.N. (Bandung Institute of 
Technology, Bandung). 

Prof. Kaye Stacey is one of Mathematics Expert Group (MEG) of PISA. I sent four problems to 
her by email on Oktober 22th, 2012, and she gave comments about the prototype I on Oktober 25th, 
2012. Her comments were: (1) these problems are all quite similar to released PISA items, so they 
have the right ‘flavour’ for PISA; (2) all of your units have only one item. Some PISA units have more 
than one item. But the items are designed to be statistically independent of each other as far as 
possible; (3) I do not know what “problem level” is. However, if it is a measure of difficulty, I think 
Batik problem will be easier than level 5; and (4) perhaps Unit 3 is “quantity” rather than change and 
relationships. However, it is often quite difficult for people to agree on the classification. 

Furthermore, I sent 13 problems to Prof. Dr. Sutarto Hadi, M.Si., M.Sc. and Prof. Dr. M. 
Salman A.N. by email also, but both of them different tools to comment our problems. Prof. Dr. 
Sutarto Hadi, M.Si., M.Sc. gave comments by writing down on the paper then sent to me, but Prof. Dr. 
M. Salman A.N. gave cooments by phone. They said that all of the mathematics problems were good, 
there were some sentences must be revised, and add the question item of the several units. They also 
said that the problems included in change and relationships content of PISA, except the “election of 
OSIS unit”. Based on three of experts comment, we revised the prototype I. 

c. One-to-One 

One-to-one and expert review conducted in parallel. One-to-one evaluation used four students 
tested at different times and places. The students were Muhammad Akbar Barokah (. MTs. Negeri 1 
Palembang), Ainayah Fadilah ( SMP Negeri 17 Palembang ), Rizky Rosari ( SMP Negeri 14 
Palembang ), and Ahmad Bagas E. ( SMP Negeri 1 Palembang). They said that the problems were 
difficult, so they didn’t answer some questions. Some of them such as Rizky and Ainayah were not 
understood in some sentences of the problems. According to experts comment and students opinion, 
we revised the prototype I became prototype II. 

d. Small Group 

Small group evaluation tested six students at different abilities. They were IX.2 grade students 
of SMP Negeri 1 Palembang. They answered the 13 questions of prototype II, but two of them 
couldn’t answered all of the questions. Almost all of them said that the problem clear enaough, they 
also said the problems does not have a variery of meanings. Based on students' answers and 
comments, the prototype II was revised became prototype III, 12 questions. 

e. Field Test 

Field test evaluation tested 29 students at different abilities. They were IX. 1 grade students of 
SMP Negeri 1 Palembang (figure 3). They answered the 12 questions of prototype III. After they 
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finished to answer the questions, they answered the questionnaire sheet, then we took seven of them to 
take interview. 



Figure 3. Students on field test 


Based on the data of questionnaire, 24 students (83%) gave positive impression about the 
mathematics problems such as figure 4. Then the interview result, six of seven students who 
interviewed said that “we are interested to solve the problems because it can help sharpen our 
thinking, so that the right-brain is functioning, not only the left-brain”. They also said that they were 
more interested answer the questions like this, than the questions like in the handbooks generally. 
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Figure 4. Students statement about all of the problems 
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Below is one of 12 questions that developed. 


! EGRANG GAME 
i 

i 

i 
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i 
i 
i 
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i 
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j The picture above is a picture of three children, namely Hendra, Sofyan, and Roni were 

! playing Egrang at the campsite. Distance (s) is the distance of child leg uses egrang in 
I 

n t 

i order, which is given the formula —= 3, where: 
i 

■ n = number of steps per child per minute, 

i t = time in minutes, 

: s = distance in meters 

! 

i Question: 

i 

j If the above formula is used to calculate the distance of Hendra, where the number of 
i Hendra steps are 30 steps per minute. Calculate how many meters the distance of Hendra 
i for 1.5 minutes? 
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Content: change and relationships 
Context : personal 

Competency : reproduction 
Prediction level : 2 

Based on the field test result, 19 students (66%) answered correctly, 2 students (7%) correct part, and 
9 students (31%) wrong answer. In the figure 5 below the example of the students answer, answer 
correctly (figure 5.a), correct part (figure 5.b), and wrong answer (figure 5.c). 



Figure 5. Students’ answer 
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CONCLUSION 

This research has been developing 12 questions of change and relationships content which was 
valid, practical, and have potential effect. The validity of the problems known based on the comments 
of three of expert on expert review stage: Prof. Kaye Stacey, Prof. Sutarto Hadi, M.Si., and Prof. Dr. 
M. Salman A.N.. Then, the practicality of the problems known based on the opinion of students on 
one-to-one and small group stage: the problems easy to use, the problems clear enough, and does not 
have a variery of meanings. The last, the problems have potential effect because 24 students (83%) 
gave positive impression about the mathematics problems. Then the interview result, six of seven 
students who interviewed said that “we are interested to solve the problems because it can help 
sharpen our thinking, so that the right-brain is functioning, not only the left-brain”. They also said that 
they were more interested answer the questions like this, than the questions like in the book generally. 
So, 12 questions of mathematics problem based PISA level on ahange and relationships content were 
valid, practical, and have potential effects to the students. 


REFERENCES 

Akker J., Nieveen, N., & McKenney, S. (2006). Design research from a curriculum perspective. In 
Akker J., Gravemeijer, K., McKenney, S. & Nieveen, N. (eds), Educational Design Research 
(pp. 67-90). London: Routledge. 

Akker J. (1999). Principles and methods of development research. In Akker J., R.Branch, K. 
Gustafson, Nieven, & T. Plomp (eds), Design Approaches and Tools in Education and Training 
(pp. 1-14). Dortrech: Kluwer Academic Publishers. 

Depdiknas (2006). Kurikulum Tingkat Satuan Pendidikan. Jakarta: Depdiknas. 

De Lange, J. (1987). Mathematics, Insight, and Meaning (Doctoral dissertation). Utrecht: OW & OC, 
Utrecht University. 

OECD. (2003). The PISA 2003 Assessment Framework - Mathematics, Reading, Science and Problem 
Solving, Knowledge and Skills . Retrieved March 2012, from 
http://www.oecd.org/edu/preschoolandschool/programmeforinternationalstudentassessmentpisa/ 
33694881.pdf. 

OECD. (2009). Learning Mathematics for Life: a view perspective from PISA. Retrieved March 2012, 
from http://www.oecd.org/pisa/pisaproducts/pisa2003/44203966.pdf. 

OECD. (2009). PISA 2009 Assessment Framework-Key Competencies in Reading, Mathematics, and 
Science. Retrieved March 2012, from http://www.oecd.org/pisa/pisaproducts/44455820.pdf. 

OECD. (2009). PISA Take the Test : sample questions from the OECD's PISA assessments. Retrieved 
March 2012, from http://www.oecd.org/pisa/pisaproducts/pisa2006/41943106.pdf. 

OECD. (2010). PISA 2012 Mathematics Framework. Retrieved March 2012, from 
http://www.oecd.Org/dataoecd/8/38/46961598.pdf. 

Stacey, K. (2010). The PISA View of Mathematical Literacy in Indonesia. Journal on Mathematics 
Education (IndoMS-JME), 2(2), 95-126. Palembang: IndoMS. 

Tessmer, M. (1993). Planning and Conducting Formative Evaluation. Philadelphia: Kogan Page. 



56 IndoMS-JME, Volume 5, No. 1, January 2014, pp. 47-56 


Van de Walle, J. A. (2007). Elementary and Midle School Mathematics , Sixth Edition. USA: Pearson 
Prentice Hall. 

Zulkardi (2006). Formative Evaluation: What, Why, When, and How. Retrieved September 2012, from 
http://www.reocities.com/zulkardi/books.html. 



